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EVALUATION OF 
WORKPLACE LITERACY 
PROGRAMS: 

A Profile of Effective Instructional 
Practices 



Lany Mikulecky 
Paul Lloyd 

Indiana University, Bloomington 



Abstract 

This study of ten groups of learners in workplace literacy programs was 
used to develop and refine a data-based model for evaluating workplace literacy 
programs. The model addresses both the programs’ need for custom-designed 
assessment and researcher/funder needs to aggregate data across several small 
programs. An analysis of the aggregated results compared curriculum and 
classroom practice with the impact of each program, and used analysis of 
variance to determine which program practices lead to success and in what 
areas. The programs were most effective at improving learners’ literacy 
performance, literacy strategies and processes, and learners’ beliefs and plans 
related to literacy. The analysis of variance allowed the development of a data- 
driven profile of thresholds for effective program practices. This profile 
suggests that these gains are linked to an environment intense with the use of 
workplace reading and writing materials, and providing regular discussion and 
feedback related to learners’ literacy processes, beliefs about personal literacy 
effectiveness, and future educational plans. 
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Introduction 



Over the three years of this project, studies of ten groups of learners in 
workplace literacy programs have been used to gather data on the impact of the 
programs in the areas of learner gains and workplace improvements. These data 
have been analyzed by comparing curriculum and classroom practice with the 
impact of each program, and using analysis of variance to determine which 
program practices lead to success and in what areas. 

Evaluation of workplace literacy programs is a relatively new area, with its 
own unique set of problems. Courses are usually short (30-40 hours), cumcula 
often focus on workplace-specific literacy tasks, and groups of learners tend to 
be small (10-15). Therefore, the kinds of assessment methods used for 
schools — standardized tests of generalized reading abilities conducted on 
hundreds of students — are inappropriate in this context. The present study uses 
a new evaluation model that was developed to assess changes in a variety of 
learner characteristics, from the ability to read workplace materials to the 
increased definiteness and detail of future plans in relation to literacy and 
education. The model includes a core of common assessment measures used by 
all programs, and a structure within which other measures can be custom- 
designed for each program. This mixture allows program assessment to focus 
upon specific literacy tasks (which change from workplace to workplace), while 
at the same time assessing program impact upon learners’ literacy practices, 
aspirations, and attitudes. The shared items and shared structure of custom- 
designed items make possible an initial attempt at merging data across programs 
so that patterns of program impact might be identified. Lack of an assessment 
approach flexible enough to address program differences and small class sizes 
has heretofore prevented most workplace literacy program analysis beyond that 
of individual programs. 

Moreover, there currently exists little systematic inquiry that might reveal 
the effectiveness of this approach or assist practitioners in determining the 
conditions under which it might be most useful. 

This report provides a brief overview of the nature of native language 
literacy instruction for adults, the reasons provided by practitioners for offering 
it, and avenues of inquiry that might further the field. The first section 
introduces readers to key characteristics of adult native language literacy 
programs around the United States, based on survey data. In the second 
section, we describe what we have found to be five key reasons for offering 
native language literacy. These are based on the literature, evidence provided by 
interviews with expert practitioners, and, when it exists, the research. In the 
third section, sever^ potential directions for further research are s’li^^e^-ted and 
various constraints and limitations associated with conducting resea' .. with this 
group of learners are discussed. 
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Background 



Only a few workplace literacy programs described in the research literature 
report any form of rigorous program evaluation or careful documentation of 
learner gains, impacts on productivity, and detailed descriptions of effective 
program practices (e.g., Haigler, 1990; Hargroves, 1989; Mikulecky & 
Strange, 1986; Philippi, 1988, 1991; Sticht, 1982; Sticht, 1995). 

The above examples are atypical. Mikulecky and d’Adamo-Weinstein 
(1991) observe that the majority of workplace literacy programs described in 
the available research literature tend to report no rigorous evaluation data. 
Many programs that do report evaluation data simply provide superficial 
information limited to surveys of learner satisfaction and anecdotal reports of 
effectiveness. Occasionally a pre and post administration of a standardized 
reading test (usually the Test of Adult Basic Education — TABE, or the Adult 
Basic Learning Examination — ABLE) provide an indication of learner gain in 
general reading ability. Only a few evaluations provide follow-up data on the 
impact of programs on learners’ job performance, retention, or earning 
power. 

Kutner, Sherman, Webb, and Fisher (1991) reviewed workplace literacy 
programs funded by the U.S. Department of Education in order to determine 
the elements of effective programs. In order to identify components of 
effective programs, the authors examined 29 of 37 projects funded by the 
National Workplace Literacy Program in order to determine which programs 
were effective and merited further examination. The authors reported that 

Due to the absence of quantitative data necessary to identify particularly 
effective projects (i.e., improved productivity, low participant attrition, or 
improved test scores), study sites were recommended to OVAE staff. These 
sites were reported by project directors to have a high retention rate. (p. 26) 

Even in federally funded workplace literacy programs, for which program 
evaluation was an expectation for receiving funding, it was not possible to 
find six programs that had been rigorously evaluated for effectiveness. 
Selection of “effective” programs was based upon undocumented reports of 
retention from program directors. 

Many evaluation procedures assumed as normal in year-long school 
programs are extremely difficult — and sometimes impossible or 
inappropriate — for workplace programs. In some programs, privacy issues 
preclude testing of any sort. In programs where testing is possible, it is rare 
for a standardized literacy test to be used as more than an initial screening 
device. In any case, it is hard to justify the use of general standardized literacy 
tests for the typical workplace situation: small classes of brief duration where 
instruction is often targeted on job-related literacy needs. 

Workplace courses tend to be brief (2-5 hours of instruction per week for 
6-8 weeks, and sometimes even less) and involve small groups of learners 
(10-15) with few similar courses taking place at the same time. It is often 
impossible to gather data on more than 15 subjects taking part in a particular 
course of instruction. Also, finding a comparable group of workers as a 
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control can present problems in a workplace where many different training 
courses are going on, because some of these are likely to overlap with the 
course being evaluated, thus making it difficult to find a group of workers who 
have not received some recent instruction that relates to the course being 
evaluated. 

Standardized reading tests measure general literacy ability and are usually 
used in schools to assess changes following daily instruction over a full school 
year. Such tests are rarely sensitive enough to assess changes over shorter time 
periods. Given the brevity of workplace literacy courses, their curriculum is 
necessarily limited and often concerns specialized workplace skills and 
competencies, rather than a broad range of general literacy skills. In this 
situation, standardized tests of general reading ability are unlikely to show much 
change in learners’ performance and are not often used by businesses. In order 
to show what learners have gained, workplace course evaluations need to use 
measures that are custom-designed to assess what has been taught. This may 
involve measuring the learners’ ability to use workplace reading materials and to 
perform their jobs more effectively. An obvious measure of the latter is a 
worker’s job productivity, but few organizations keep data of this kind on 
individuals — the smallest unit tends to be the work team. The most practical 
measures of individual workplace competence are employee rating scales 
(usually filled out by supervisors) and job-related reading scenarios. And both 
of these need to be custom-designed for particular workplaces. 

However, solving the problem of assessment in one workplace generates 
problems for the researcher who wants to compare gains made at different sites 
and enlarge the sample size in a study by amalgamating results across 
programs. The workplace-specific measures, while providing useful 
information on each workplace, are necessarily different from each other, and 
introduce difficulties of comparison across programs. Given these difficulties, 
the National Center on Adult Literacy (NCAL) Workplace Literacy Impact 
project was designed to pilot and refine an evaluation model that employed a 
broad definition of literacy, that could be sensitive to the differing literacy 
demands of a variety of workplaces, and that allowed for comparisons of 
results across programs. The model was based on an overall structure 
consisting of some assessments common to all workplaces and of a framework 
within which program personnel could custom-design instruments suitable to 
their instruction and workplaces, while keeping an element of comparability 
with the assessments used at other programs. 



Conceptual Framework 



The conceptual framework for this workplace literacy impact assessment 
model is based upon a broad understanding of literacy. In addition to assessing 
improved performance with a variety of literacy tasks, the model also assesses 
changes in life-style related to literacy and changes in learners’ self-perceptions 
and aspirations in relation to literacy. The model owes much to ideas presented 
and developed by Lytle (1990) in long-term case studies of changes in adults 
experiencing success in adult literacy programs. It was also influenced by ideas 
about perceptions of self-efficacy developed by Bandura ( 1986) and others who 
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have studied why some individuals outperform others who have similar tested 
abilities. 

Lytle (1990) has suggested that performance measures (tests and 
exercises) miss a good deal of important information about adult literacy 
learning. Lytle has examined the literacy growth of adults enrolled in adult 
literacy programs. Extensive observations, interviews, and learner journals 
were used to track changes that took place as learners spent a year in adult 
literacy instruction. Lytle found that in addition to gains in literacy skills, 
adults tend to make changes in their beliefs, behaviors, and aspirations. These 
changes are intertwined and seem to influence each other. For example, new 
understanding about the process of how reading and writing work may 
influence beliefs about what the learner can accomplish. Changes in 
aspirations resulting from an increased sense of effectiveness sometimes lead 
to increased practice and increased competence. Lytle suggests several 
dimensions that constitute a fuller understanding of adult literacy and adult 
literacy growth. These dimensions are learners’ beliefs about literacy and 
themselves, learners’ literacy practices, the literacy processes employed by 
learners while reading, and the plans a learner has that may involve literacy 
use. Programs and assessments that focus merely upon performance are likely 
to miss the complex, intertwined mixture of changes in (a) self-perception, (b) 
literacy life-style, (c) understanding of the literacy process, and (d) hopes and 
aspirations, all which are necessary elements in improving and sustaining 
literacy performance. 

Lytle’s conceptual framework has been adapted to the present workplace 
literacy project in order to test the importance of these aspects of adult 
learning: beliefs, practices, processes, and plans, and in order to seek out 
ways to enhance learning. Information about these dimensions of learners’ 
literacy were gathered using a combination of questionnaire items, interview 
questions, and requests that learners explain their literacy strategies or 
processes while involved in simulated job tasks. 

Bandura (1986) has written extensively about the influence upon 
performance of self-perception of one’s effectiveness. His concept of 
perceived self-efficacy has been used to examine the performance of children 
and adolescents in school, as well as to examine the performance of adults in a 
variety of life situations from athletic competitions to success in substance 
abuse treatment programs to academic and career success. Results of several 
studies with adults experiencing phobic reactions led Bandura to note, “People 
who are burdened by acute misgivings about their coping capabilities suffer 
much distress and expend much effort in defensive action’’ (Bandura, 1986, 
pp. 425-426). Adult literacy researchers (Bean, Partanen, Wright, & 
Aaronson, 1989; Van Tilburg & DuBois, 1989) have noted such distress 
leading to poor performance and abandoning of programs by adults receiving 
literacy instruction. Perceived self-efficacy (based upon accurate feedback) is 
particularly important in relation to adult literacy learning. Adults with high 
and accurate perceptions of their personal literacy abilities tend to try harder, 
to continue in the face of obstacles, and to succeed more often than learners of 
comparable tested ability but lower senses of personal effectiveness with 
literacy. While learners with low senses of personal effectiveness tend to 
subvert their own efforts with self-doubt and excuses for quitting, learners 
with higher senses of effectiveness often perform successfully and continue to 
learn as a result of persistence. Some other studies (Shunk & Gunn, 1985; 
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Shunk & Rice, 1987) have also demonstrated that interventions that are 
designed to improve children’s perceptions of their effectiveness with literacy 
strategies are associated with improved literacy performance. In the workplace 
literacy program impact model discussed in this study, Lytle’s conception of 
learners’ beliefs about literacy is expanded to focus more specifically upon 
beliefs about personal effectiveness with literacy. 



Method 



The purposes of this study were (a) to gather data on the effectiveness in a 
variety of areas of a number of workplace literacy programs, and (b) to compare 
those results with the nature of the instructional practices in the programs. This 
comparison was intended to reveal specific connections between teaching and 
learning practices and learner gains in the several areas related to workplace 
literacy in our workplace literacy impact model. The focus of the model’s 
assessment is on changes in learners’ beliefs about personal effectiveness with 
literacy, changes in learners’ literacy practices, learners’ literacy improvement 
with general and workplace materials, and changes in learners’ goals. 

I Project personnel worked with on-site coordinators at each company to 
develop instruments using model guidelines. These instruments were designed 
to assess the effect of the literacy courses on the learners’ own literacy 
behavio’^s and on their work competency. For the first of these, Lytle’s literacy 
model (Beliefs, Practices, Process, Plans) was used as a basis for structuring 
measurement of learner change (Lytle, 1990). During structured pre- and post- 
interviews, learners were asked about themselves and their abilities in relation to 
literacy, aliout their reading and writing practices, about how they read print 
materials, and about their future educational plans. Pretest data were gathered at 
the start of each course and posttest data toward the end. All data were then sent 
in to the project for analysis. In addition, discussions with instructors, 
classroom observations, and analysis of curricular materials provided a 
foundation for rating instructional emphases at each worksite. 

In order to analyze the results of the ten groups of learners, rating schemes 
were constructed to summarize the characteristics of each course and its 
curriculum, in such areas as emphasis on workplace examples, reading and 
writing intensity, and discussion of literacy beliefs and plans. Comparisons 
between these course-characteristic ratings and learner-gain scores have been 
used to identify program characteristics that produce learner gains, and analysis 
of variance has been used to indicate workplace literacy program practices that 
lead to success within the various components of the workplace literacy impact 
model. 

Populations: Locations and Subjects 

The ten groups of learners (« = 181) attended workplace literacy programs 
at six companies and completed pre- and post-assessments. No program 
administered pre and post standardized reading tests, although two programs 
used such tests during initial screening. Reading ability ranges in these 
programs were similar to those in high school classes (i.e., from high 
elementary school levels to beginning college levels). Age ranges for students 
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were from the low 20s to over 60, with the majority of students between 25 
and 45 years of age. The typical student had been with the cun'ent employer 
for more than a decade. Details of the companies and instructional groups 
follow. 

Site #l is a large manufacturing plant, where three courses were conducted: 
Technical Preparation — a 6- week, 7-hours-per-day course designed to 
prepare employees for subsequent technical training; 

GED — meeting for 4 hours per week over 6 weeks; and 
ESL — meeting for 8 hours per week over 6 weeks. 

For the first of these, there was a control group made up of workers who had 
not yet begun the Technical Preparation course. Each of the four groups 
consisted of 12-15 employees. 

Site #2 is a women’s prison, where correctional officers and other staff 
attended the following course; 

Report Writing — 28 staff were in class 3 hours per week for 13 weeks, 
learning how to improve the quality of the reports that they need to write 
as an integral part of their work; and 

Promotions Support — 9 staff spent 3 hours per week for 7 weeks in 
enhancing the skills that they need to apply for promotion, including 
preparation for a promotion test. 

Site #3 is a small insurance company, where a group of 20 learners was 
improving skills in job-related tasks connected with reading and writing. 
Some attended class for 20 hours and completed a similar amount of work 
outside class, while others worked with self-study packages supplemented 
by individual help sessions with an instructor. All were engaged in their 
studies for about 40 hours in total. 

Site #4 is a hospital, where 19 service employees attended a computer-based 
writing course for 20 hours, plus up to 10 more hours of out-of-class 
practice. They were learning the basics of word-processing and, at the 
same time, working on their study and writing skills, particularly for 
writing memos. 

Site #5 is a large gasket-maker, where data weie gathered on 10 learners in 
basic reading and writing courses. These learners attended two courses in 
sequence, for a total of 50 hours. They practiced reading workplace 
materials such as newsletter articles, procedure manuals, and productivity 
graphs. They were taught how to fill out forms, complete logs, and take 
notes at a meeting. 

Site #6 is a manufacturer of electric motors, where reading skills enhancement 
courses were conducted for the employees. The goal was to raise the 
reading level of all employees to at least the eighth-grade level. Data were 
gathered on 33 learners, who attended class for 30 hours. They were 
taught a variety of reading strategies, such as questioning, summarizing, 
reviewing, skimming, and scanning. 
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Instruments 



Data were collected for each learner before and after each course, or at 
suitable intervals for ongoing courses. In a one-on-one structured interview, 
information was gathered on learners’ beliefs about literacy in general and their 
own literacy effectiveness in particular. The interviews also focused on literacy 
practices and learners’ plans for 1, 5, and 10 years ahead. In these three areas, 
the same questions were asked at ^1 programs. To measure job-specific literacy 
processes and abilities, all site coordinators participated in the development of 
job-related scenarios based on workplace reading materials. Also, the site 
coordinators provided information about curriculum and classroom practices, so 
that ratings could be developed to compare the characteristics of the different 
courses. 

The interview and job-related scenarios used in this analysis were just part 
of a wider range of instruments that form part of the full evaluation model 
developed in this project. Other instruments include questionnaires on 
frequencies of literacy practices and employee job performance ratings. (For a 
full set of instruments, see Appendix C.) 

INTERVIEW: General 

An interview protocol was devised to cover the four aspects of Lytle’s 
model. Three of these aspects — practices, beliefs, and plans — were tested using 
open-ended questions that applied to all learners at all sites. Concerning literacy 
practices, learners were asked about their reading and writing, both at work and 
away from work. A typical question is the following: 

“Tell me the sorts of things you read and write on the job during 
a normal week.’’ 

This was followed up with a non-directive prompt (“Can you give me more 
examples?’’), which was repeated several times until the learner had no more to 
add. Concerning their beliefs, learners were asked to describe their perceived 
level of literacy, and what it might become in future. One set of questions was 
the following: 

“How good do you consider yourself to be at reading and 
writing? What makes you think so?’’ 

Finally, learners were asked about their future plans, for 1, 5, and 10 years 
ahead, and how they saw reading and education as part of those plans. For 
example: 



“Now I’d like to ask you about your plans. Explain how you see 
reading and education as part of these plans. What are your 
plans for the next year?’’ 

Job-Related Scenarios 

The fourth aspect of Lytle’s framework — reading process — was assessed 
using site-specific job-related scenarios. Three different job-related items (i.e., a 
newsletter article, a graph, and a chart or procedure) were selected with the 
advice of the site coordinators. The subjects were presented with these 
workplace materials, and asked to go over them and describe how they read 
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(i.e., reading strategies, areas of focus, thoughts while reading). All learners 
were asked essenti^y the same question in each scenario about how they read 
the materials: 

“ I am going to show you a newspaper article about your 
industry. I want you to explain to me how you would read it. 

There are many ways to read. People look at different parts of 
a page and think about different things when they read. What 
would you do first, then next, then next?” 

After reading, they were asked to answer factual, inference, and 
application questions about the specific contents of the reading materials. 

Data Gathering Procedures 

The interview and job scenarios were conducted by a researcher one-on- 
one with a learner. The researcher asked each question and made notes on the 
learner’s responses, pausing long enough to obtain a considered answer and 
using standard non-directive prompts and probes to elicit a more extensive 
response. The time taken for each individual interview was in the range 20-30 
minutes. These interviews were conducted as pretests at the start of each 
course end as posttests at the end. 

Data Analysis Techniques 

Data analysis was of two types: category and holistic. Quantifiable 
interview responses were recorded and analyzed statistically. Responses to 
open-ended interview questions were recorded, and then methods of analysis 
were developed to fit the nature of the responses. 

For some open-ended interview questions, categories of responses were 
allowed to emerge from the data. These categories were then used to label 
subject comments. When category refinement flowed for acceptable levels of 
interrater agreement (90% or higher), category responses were recorded and 
statistically analyzed. For other open-ended interview questions, a holistic 
comparison was made between pretest and posttest responses, and the change 
was rated as positive, neutral, or negative. As with the category schemes, the 
criteria for assessing this change emerged from the data, and the application of 
the scheme was subject to the same levels of acceptable interrater agreement. 

Examples of both category and holistic rating schemes arose in connection 
with the interview question: “How good do you consider yourself to be at 
reading and writing? What makes you think so?” Responses to this open- 
ended question nearly always included spontaneously some kind of self- 
rating, using words such as average, very good, below average, and poor. 
These were categorized from lowest to hipest on a scale of 1-5, to produce a 
score for each self-rating. In addition, a holistic rating was applied to the full 
response, in which change from pretest to posttest was judged as positive, 
neutral, or negative according to the reasons given by the subject for the 
reported self-image. An example to illustrate this process follows: 

Pre\ I’m not very good at reading and writing — not much 
education. 
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I Post: I’m about average. I’m not stupid. I can read and write — I 

can figure things out. 

For this response, the self-ratings were 2 for the pre-interview and 3 for the 
post-interview, and the holistic rating was positive because of the change from 
low to higher self-esteem. 

The other area in which a holistic rating was used related to subjects’ plans 
for the future. These were judged on their de^'initeness and detail, as the 
following example illustrates: 

Pre: I don’t know. 1 might go to tech school in a couple of 
years. 

Post: I'm going to school — four nights a week in the fall. 

This too was rated positive, because of the change from vague intentions to 
a definite decision. 

For any of the responses which resulted in numerical scores (e.g., materials 
read during a week at work or scores for scenario questions), statistical tests 
were applied to the set of scores for each group of subjects. Pre- and post- 
assessments were compared for the individuals in a course using a paired- 
sample Mest, in order to detect gains brought about by the program. In 
addition, for the holistic change scores, the allocation of values +1,0, and -1 to 
positive, neutral, and negative allowed the use of a one-sample r-test to 
determine if the changes were significantly different from 0. In all cases, as the 
tests were of no difference versus improvement, the statistical tests were one- 
tailed. 

Course Characteristics Ratings 

For the analysis of variance described below, it was necessary to construct 
rating schemes to summarize the instructional characteristics of each course and 
its curriculum. These covered such areas as emphasis on workplace examples, 
reading and writing intensity, focus on improving literacy processes and 
strategies, and discussion of literacy beliefs and plans. The site coordinators 
provided syllabi, assignments, and other curriculum documents, together with 
descriptive observational information about the teaching methods of the teachers 
at their sites. Using these materials, the ratings were completed by two 
researchers, who then compared results, discussed differences in ratings when 
these occurred, referred back to on-site coordinator notes and curriculum 
documents as necessary, and then reached consensus on the ratings. 

Combining data across programs is always problematical. In an ideal world, 
data would be combined only for students receiving identical instruction and 
being assessed with identical measures. The second of these is easier to achieve 
than the first in any situation, but — in this workplace study — neither is strictly 
tme. However, all learners did receive instruction in literacy aimed at improving 
their abilities in reading and writing, which argues for at least as much 
commonality as, for example, in a study of high school seniors and their 
academic performance. 

In fact, the programs and courses that took part in this project had much in 
common. All programs were based in workplaces and set out to increase the 
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literacy skills of those workers attending. All addressed increasing the literacy 
and communications competence of the learners in order to help them deal 
with practical situations in the workplace and everyday life. All classes were 
small (10-15 learners) and included much individual attention on particular 
learner problems. Also, within the wider context of schooling, all courses 
were brief. Most were of 20-50 hours duration, and the one exception 
included 200 hours of instruction, which corresponds to only 6 or 7 weeks of 
high school. 

Also all learners were assessed within the framework of the evaluation 
model. This as.sessment included many elements common to all programs, 
and those measures — the job-specific scenarios — that differed in topic from 
program to program were all constmcted to the same pattern. The three 
custom-designed scenarios for each workplace included a range of factual, 
inference, and application questions, and included prose, document, and 
quantitative materials. Thus, a compromise was achieved, which allowed site 
specific assessment appropriate to each program and, at the same time, made it 
arguably feasible to aggregate data across programs in order to identify 
patterns. 

As with most instmction, there was enough variation among the programs 
to allow an analysis of the effects of those differences on learner gains. 
Although all programs were quite brief compared to K-12 schooling and all 
addressed workplace literacy goals, the content of the materials varied. Some 
concentrated very specifically on workplace skills and materials, while others 
employed a greater proportion of more general literacy materials. As with 
instruction in schools, there were differences in the teaching styles employed: 
from a mix of lecture, learner practice, and discussion to more structured 
learning modules with occasional conferences with the teacher. The 
percentage of class time spent practicing reading and writing or in discussion 
of reading strategies or the relevance of instruction to future goals also 
differed somewhat from class to class. In order to encapsulate these 
differences, six factors were chosen to characterize the various courses. They 
cover three aspects of course practices: 

• Time — total instructional time and time spent on reading and 

writing, 

• Topic — workplace and home/family orientation of instmction, and 

• Talk — discussion of literacy beliefs and plans and of reading and 

writing processes. 

The six factors are listed below together with abbreviated labels for their 
use. . 
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Factors for Rating Course Characteristics 



1 . Hours 

2. RAVr 

3. Work 

4. Home 

5. DiscB/P 

6. DiscProc 



Instructional time in hours — course meetings + homework 

Reading/writing intensity — 0-5 scale with descriptors* 

Workplace orientation — 0-5 scale with descriptors 

Home/family orientation — 0-5 scale with descriptors 

Discussion of literacy beliefs and plans — 0-5 scale with descriptors 

Discussion of reading and writing processes — 0-5 scale with 
descriptors 



(* For details of all descriptors, see Appendix A.) 

Although anchor points were different for the various scales, the two 
examples given below are typical. 



3. Workplace Orientation 
Curriculum and materials: 

0 have no direct connection with the workplace 

1 use workplace examples occasionally 

2 use workplace examples sometimes (20-30% of time) 

3 use workplace examples much of the time (50-b0% of time) 

4 are connected mainly to the workplace (70-80% of time) 

5 are connected entirely to the workplace (90-100% of time) 



5. Discussion of Literacy Beliefs and Plans: 

0 does not occur in this course 

1 occurs occasionally/incidentally in tliis course 

2 occurs as a deliberate part of this course 

3 occurs moderately often in this course (every other session) 

4 occurs often in tliis course (most sessions) 

5 occurs very often in this course (every session) 



The three aspects of Time, Topic, and Talk were chosen to characterize the 
courses because of their importance in the workplace context. Most workplace 
literacy programs are brief (20-50 hours) and therefore need to be targeted on 
very specific goals. Thus, total time is of the essence in determining what can 
be achieved, as is the amount of time spent practicing reading and writing. 
Because of the limited nature of transfer from one area of application to another, 
it is also important to know what kinds of topics are used to teach literacy in 
these programs. The use of workplace examples is more likely to carry over to 
job-related scenarios than the use of more general material. Finally, due to the 
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limited time available, learners are unlikely to make connections on their own 
about generalizable reading strategies or about self-esteem and future plans. 

The six course characteristic scales were then applied to each course using 
syllabi, assignments, and other curriculum documents, together with 
classroom observations and descriptive information about teaching methods. 
The results of this process are contained in Table 1. 



Table 1 

Ratings of Course Characteristics 



FACTOR HOURS 

COURSE 


R/WR 


WORK 


HOME 


DISCB/P 


DISCPROC 


Site^l: Automotive 


Tech Prep 


200 


3 


2 


1 


2 


2 


Control 


0 


0 


0 


0 


0 


0 


GED 


24 


5 


1 


1 


1 


1 


ESL 


48 


2 


2 


2 


3 


2 


Site^2: Women *s prison 


Report Writing 


40 


4 


4 


0 


2 


5 


Promotions 


20 


4 


5 


1 


3 


2 


Site^3: Insurance 


40 


5 


4 


1 


1 


4 


Site^4: Hospital 


20 


5 


1 


1 


2 


3 


S itei¥5: Gasket-maker 


50 


4 


4 


I 


1 


3 


Site^6: Electric motors 


30 


3 


1 


2 


1 


5 



For one of the rating scales listed in Table l(Home/family orientation), 
nearly all courses were rated 0 or 1, showing that this area did not play a 
significant part in the curricula of the courses under consideration. Therefore, 
this factor is not used in the later analysis. 

In order to investigate any possible cormections among the rating scales as 
causes for learner gains, the ratings were applied to each learner in the sample 
according to the course attended, and the correlations between the rating scales 
were calculated on a sample of 181 learners. Several of these correlations 
were statistically significant, but none were very large. For example, the 
highest correlations were (a) 0.604 between Reading/writing intensity and 
Workplace orientation, (b) 0.409 between Workplace orientation and 
Discussion of beliefs and plans, and (c) 0.351 between Reading/writing 
intensity and Discussion of reading processes. 

Even so, this suggested that factor analysis should be used to establish 
whether some of the course characteristics could be combined into more 
useful constructs. The results of this process, however, revealed no clear 
grouping of the course characteristics within the factors generated, so it 
appears that each of the rating scales used in this study involves a separate 
instructional factor in the courses under investigation. 
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A second approach toward simplifying the picture of course characteristics 
involved trying to establish course profiles, into which several courses would 
fit. Differences in instructors’ styles across the course characteristics did not 
allow for any clear generalization in this direction. 

Learner Gains 

The course characteristic ratings were then compared with the gain scores 
(see below) of each individual learner in all of the courses pooled together, in 
the areas evaluated by the learner interview. Those areas are as follows: 

1. Practices at work 

2. Practices away from work 

3. Reading process 

4. Scenario performance 

5. Beliefs (literacy self-efficacy) 

6. Plans 

(For details of the calculation of these scores, see Appendix B.) 

For Items 1-3, each learner’s gain score is the difference between a pretest 
and a posttest score. Item 4, scenario performance, is based on the sum of 
scores on site-specific reading scenarios; because topic materials were different 
for each program, the scores on each scenario were standardized before being 
combined with the other scenarios. For Items 5 and 6, the gain scores are based 
on a direct comparison of pretest and posttest interview responses. With the 
exception of Item 4, each area of learner gain was assessed using the same 
questions for all courses at all programs. 

Analysis of Variance 

The technique of analysis of variance (ANOVA) was used to compare the 
gains of those groups of learners having high ratings on particular course 
characteristics with those having low ratings on that characteristic. This 
indicates which course characteristics play important roles in determining 
whether or not learners make gains in each area. In addition, patterns across 
courses can suggest the threshold level of the activity related to that 
characteristic (i.e., use of workplace materials or discussion of literacy 
processes) that is required to produce measurable learner gains in various areas 
of the workplace literacy program impact model. Although this process is only 
an initial attempt to identify patterns across programs, it can allow some 
tentative insights about a desirable mixture of course activities that will produce 
learner gains in a variety of areas related to workplace literacy competence. 

The division between high and low ratings for each characteristic was 
detemiined by first carrying out analyses of variance with all characteristic 
ratings separate. Where inspection showed a change in the level of learner gain 
at a particular point on the characteristic scale, this was investigated further, in a 
subsequent analysis of variance, by combining all the groups of learners below 
that point and comparing them with all the groups of learners above that point. 
The authors investigated all possible pairings of course characteristics with 
areas of gain, but the results set out below show only those for which there is a 
statistically significant difference between the high- and low-rated groups. 
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Before proceeding with the details of the divisions, a somewhat surprising 
observation must be noted. The division bt^^een high and low ratings for 
each characteristic that is paired with more than one area of gain occurs nearly 
always at the same point on the characteristic scale, regardless of the area of 
gain being analyzed. For example, the characteristic Discussion of literacy 
beliefs and plans is paired with the three areas of gain Scenario performance, 
gain in Beliefs about one’s own literacy abilities, and gains in the specificity 
and detail of future educational Plans. In each case, the change in the level of 
learner gain occurred at the same point of division between low and high 
ratings of Discussion of literacy beliefs and plans, suggesting that thresholds 
of activity may indeed be relevant. 

The divisions for all course characteristics are shown in Table 2. 



Table 2 



High and Low Ratings of Course Characteristics 



Course characteristic 


Division 


Rating 


Description 


Number 

of 

subjects 


Instructional time in 


low 


0-50 


up to 50 hours 


165 


hours 


high 


200 


over 50 hours 


14 


Workplace orientation 


low 


0, 1 


little connection with 


85 




high 


2-5 


workplace 

use workplace examples at 


95 


Discussion of literacy 


low 


0. 1 


least 20-30% of time 
at most occasional/ 


95 


beliefs and plans 


high 


2. 3* 


incidental 

deliberate part of course 


85 


Discussion of reading and 


low 


0. 1 


at most occasional/ 


27 


writing processes 


high 


2-5 


incidental 

deliberate part of course 


153 


Reading/writing intensity 


low 


0-3 


up to 50% of lime 


65 


1 


high 


4, 5 


over 50% of time 


96 


Reading/writing intensity 


low 


0-4 


up to 70% of time 


127 


2 


high 


5 


over 70% of lime 


53 



* ratings of 4 and 5 did not occur 



Results 



The areas of learner gains described above were compared with the course 
characteristic ratings, and cut-points in the rating scales were identified at 
which the amount of learner gain changed significantly. Using these divisions 
into high and low ratings on the course characteristics, we have the 
statistically significant ANOVA results set out below. In each case, the 
following information is given: first, the area of gain, the course 
characteristic, and the level of significance for the difference between the low 
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and high groups (p =xxx); then, for each group, a description of the division 
into low and high ratings, the number of subjects (n), the mean gain and 
standard error (s.e.) of that mean, and the resultant significance of gain for that 
group; and finally, the standard deviation (s.d.) of each group. This last is to 
verify that the variances of the low and high groups are always approximately 
equal — a necessary condition for the validity of analyses of variance. The 
findings from each case are listed: 



Practices Away From Work divided according to Instructional time in hours 

(p = 0.0120) 



Division 


n 


Mean gain 


S.e. of mean 


Significance of gain | S.d. of gain { 


low (0-50 hr) 


165 


0.121 


0.180 


n.s. ip = 0.2507) 


2.311 


high (200 hr) 


14 


1.786 


0.764 


sig. (p = 0.0181) 


2.860 



• The learners with 200 instructional hours (the Tech Prep group at Site 
#1) made significant gains in reading practices away from work, but 
all the other learners with 50 hours or less, did not. 



Reading Process divided according to Reading/writing intensity 2 (p = 
0.0003) 



Division 


n 


Mean gain 


S.e. of mean 


Significance of gain 


S.d. of gain j 


low (<70% of 
time) 


127 


0.945 


0.316 


sig. ip = 0.0017) 


3.560 


high (>70% of 
time) 


53 


3.491 


0.755 


sig. ip < 0.0001) 


5.497 


• Although both the high- and low-rated learners made significant gains 
in increased self-reported sophistication in the reading processes that 
they wc 'dd employ, the learners who spent over 70% of their course 
time reading and writing had a mean gain that was over three times 
that of the other learners. 


Scenario Performance divided according to Workplace orientation (p = 
0.0179) 


Division 


n 


Mean gain 


S.e. of mean 


Significance of gain 


S.d. of gain | 


low (<20% of 
time) 


85 


0.858 


0.214 


sig. ip < 0.0001) 


T976 


high (>20% of 
time) 


95 


1.622 


0.234 


sig. ip < 0.0001) 


2.276 J 



• Although both the high- and low-rated learners made significant gains 
on job-related scenario comprehension questions, the learners who 
used workplace examples in class at least 20-30% of the time had a 
mean gain on the reading scenarios that was nearly twice that of the 
other learners. 



Scenario Performance divided according to Discussion of beliefs and plans 

{p < 0.0001) 





Division 


R 


Mean gain 


S.e. of mean 


Significance of gain 


S.d. of gain 




low 

(incidental) 


127 


0.945 


0.316 


sig. (p - 0.0017) 


3.560 




high 

(deliberate) 


53 


3.491 


0.755 


sig. ip < 0.000 1) 


5.497 



• Although both the high- and low-rated learners made significant gains 
on job-related scenario comprehension questions, the learners who 
had discussions of literacy beliefs and plans as a deliberate part of 
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their course had a mean gain on the reading scenarios that was 
nearly three times that of tlie other learners. 



Scenario Performance divided according to Discussion of reading processes 
ip = 0.0008) 



Division 


n 


Meon gain 


S.e. of mean 


Significance of gain 


S.(L of gain 


low 

(incidental) 


27 


-0.015 


0.331 


n.s. (p = 0.4827) 


1.718 


high 

(deliberate) 


153 


1.486 


0.175 


sig. {p < 0.0001) 


2.165 



• The learners who had discussions of reading and writing processes 
as a deliberate part of their course made significant gains on the 
reading scenarios, but the other learners did not. 



Beliefs divided according to Discussion of literacy beliefs and plans (p < 
0 . 0001 ) 



Division 


n 


Mean gain 


S.e. of mean 


Significance of gain 


S.d. of gain 


low 

(incidental) 


92 


0.120 


0.104 


n.s. ip = 0.1255) 


0.993 


high 

(deliberate) 


85 


0.812 


0.095 


sig. (p < 0.0001) 


0.880 



• The learners who had discussions of literacy beliefs and plans as a 
deliberate part of their course made significant gains in the area of 
beliefs and perceived self-efficacy in relation to literacy, but the 
other learners did not. 



Plans divided according to Reading/writing intensity 1 (p = 0.0355) 



Division 


n 


Mean gain 


S.e. of mean 


Significance of gain 


S.d. of gain 


low (<20% of 
lime) 


85 


0.858 


0.214 


sig. (p< 0.0001) 


1.976 


high (>20% of 
time) 


95 


1.622 


0.234 


sig. (p< 0.0001) 


2.276 



• The learners who spent over 50% of their course time reading and 
writing made significant gains in the area of specific, detailed future 
plans, but all the other learners did not. 



Plans divided according to Workplace orientation (p = 0.0111) 



Division 


n 


Mean gain 


S.e. of mean 


Significance of gain 


S.d. of gain 


low (<20% of 
time) 


69 


0.174 


0.169 


n.s. (p = 0.1535) 


1.403 


high (^0% of 
time) 


92 


0.750 


0.147 


sig. (p < 0.0001) 


1.411 



• The learners who used workplace examples at least 20-30% of the 
time had a mean gain in the area of plans, but all the other learners 
did not. 



Plans divided according to Discussion of literacy beliefs or ■' plans (p < 
0 . 0001 ) 



Division 


n 


Mean gain 


S.e. of mean 


Significance of gain 


S.d. of gain 


low 

(incidental) 


77 


0.039 


0.157 


n.s. (p = 0.4026) 


1.381 


high 

(deliberate) 


84 

1 1 


0.929 

i 1 


0.147 
1 


sig. (p < 0.0(K)1) 

L 


1.351 



26 
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• The learners who had discussions of literacy beliefs and plans as a 
deliberate part of their course made significant gains in the area of 
plans, but the other learners did not. 



Plans divided according to Discussion of reading processes (p = 0.0141) 



Divisicn 


n 


Mean gain 


S.e. of mean 


Significance of gain 


S.d. of gain 


low 

(incidental) 


27 


-0.111 


0.222 


n.s. {p = 0.3106) 


1.155 


high 

(deliberate) 


134 


0.627 


0.126 


sig. (p < 0.0001) 


1.454 



• The learners who had discussions of reading and writing processes 
as a deliberate part of their course made significant gains in the area 
of plans, but the other learners did not. 



It can be seen that, in all of the cases set out above, there is a significant 
difference between those groups with low ratings and those with high ratings. 
Also, those groups with high ratings made significant gains, compared with no 
gains or smaller gains for those with low ratings. 

DISCUSSION OF RESULTS 

The classroom practices related to each of the characteristic rating scales 
considered above will now be examined, and these practices will be compared 
with the areas of gain that are connected with them. The course characteristics 
are Instructional time in hours. Reading/writing intensity. Workplace 
orientation. Discussion of literacy beliefs and plans, and Discussion of reading 
and writing processes (see Appendix A for definitions). 

Instructional time in hours is a factor only in relation to learner gains in 
reading practices away from work. This result depends on the fact that the Tech 
Prep group at Site #1 (who received 200 hours of instruction) made gains in 
this area and other learners did not. Although this could be an aberration caused 
by this one group, it may be a threshold effect due to the much greater time that 
these learners spent in class. Gains in practices could be longer term in taking 
effect than those in other areas. 

Reading/writing intensity (i.e., percentage of class time spent reading and 
writing) appears to be related to gains in learners’ reporting of more 
sophisticated reading processes, and more detailed and specific educational 
plans. The cause for the positive impact on forming detailed educational plans is 
not clear, but there is an obvious link between the practicing of reading and 
writing, and gains in the sophistication of reading techniques. From the analysis 
of variance, it seems that quite a high level of reading/writing intensity is 
required to achieve this; Gains remain relatively small in classes that report less 
than 70% of instructional time dedicated to reading and writing. In classes with 
70% or more of instructional time allocated to reading and writing practice, 
gains are considerably higher. 

Workplace orientation of materials and instruction is related to learner gains 
in answering comprehension questions on workplace scenarios and in forming 
detailed educational and literacy -related plans. The gains in plans may be due to 
a workplace orientation making explicit the link between future planning and job 
advancement. The connection between use of workplace materials in the 
classroom and success with assessments based on workplace materials is more 
direct: It is simply “test what you teach.” For both areas of gain, the level of use 



NATIONAL CENTER ON ADULT LITERACY 



17 




of workplace materials need not be very high — 20-30% of course time — but it 
does need to be more than incidental, built into the stmcture of the course. 

Classroom discussion and feedback related to perception of personal 
effectiveness with literacy (i.e., beliefs) and future educational plans (i.e., 
plans) are related to learner gains in answering comprehension questions on 
workplace scenarios, and in articulating learners’ own literacy-related beliefs 
and plans. Improvement in personal sense of literacy effectiveness and more 
detailed educational plans are directly connected to feedback and discussion of 
these subjects in class. Providing feedback and talking about learners’ beliefs 
and plans does appear to produce changes in their attitudes. The 
comprehension gains on the scenarios may be related to an increase in 
motivation to succeed, caused by raised levels of belief in oneself (self- 
efficacy) and by a more definite view of where education might lead. Bandura 
(1986) has shown that the willingness to attempt and persist with a task is 
related to “perceived self-efficacy” — the belief in self and ability to succeed, 
which can give a learner the confidence to continue in the face of difficulties. 
If these ideas are made explicit in classroom discussions and if learners are 
provided with feedback about their growing effectiveness, these learners seem 
more likely to develop the persistence that will give them a better chance of 
performing well at new literacy tasks. From the analysis of variance, it 
appears that it is important for such discussions to be an integral part of the 
learners’ instructional experience, rather than incidental. This does not mean 
including discussions of literacy beliefs and plans in every course session, but 
it does mean that these discussions should be frequent enough to inform the 
ongoing process of education. Courses where such discussions were planned 
demonstrated much larger gains than those where discussions of literacy 
beliefs and plans were left to chance. 

Classroom discussion and feedback related to reading and writing 
processes appear to affect learner gains in answering comprehension 
questions on workplace scenarios and in forming detailed educational plans. 
The cause for the impact on learners’ plans is not clear, but discussing literacy 
processes has an obvious connection with the ability to understand and 
answer questions about new reading materials. From the analysis of variance, 
it seems that such discussions need to be an integral part of the learners’ 
course, rather than incidental, but do not need to occur in every course 
session. It is sufficient for these discussions to happen often enough to inform 
the ongoing process of reading and writing. 

Taking these course characteristics together, it seems reasonable to 
hypothesize the following structure for a workplace literacy course to be a 
success in a wide variety of areas. It should include a large proportion of time 
when learners practice reading and writing (70-80% of course time) and a 
substantial proportion of workplace examples (about 30% of course time). 
Integrated into this, but without detracting from the reading and writing 
practice time, there should also be planned regular discussion both of learner 
beliefs and plans concerning literacy and of reading and writing processes. 
With such a mix, the results above suggest that learners ought to make gains 
in their reading abilities and sophistication of strategy knowledge, in their 
beliefs in their own literacy effectiveness, and in their abilities to plan for a 
future connected to literacy and education. In addition, for longer running 
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courses (i.e., 200 hours), changes in learners’ everyday literacy practices may 
also be expected. 



Conclusion 



This three-year study was designed to determine the feasibility of 
developing a workplace literacy assessment model that could produce 
information both of use to local program providers and capable of aggregation 
across programs. 

The study has demonstrated the feasibility of implementing the assessment 
model at a variety of sites. The inclusion in the model of a mixture of common 
assessment instruments used at all programs and a framework for developing 
customized instruments for each program has allowed for the variations in job 
materials used and in skills being taught at the different workplaces. Also, 
employing a broad conception of literacy makes it possible to be sensitive to 
program variations and to demonstrate program gains even when evaluation 
constraints are very limiting. 

The feasibility of aggregating data across sites has been demonstrated, but 
this must be viewed with a good deal of caution. Although the variation in 
instructional approaches in workplaces may not be much greater than the 
variation found in schools, this variation must be weighed in interpreting the 
results of aggregating data. Similarly, one must consider whether differences in 
adult ages and work site experiences are so great as to compromise the wisdom 
of aggregating data. On the other hand, teachers in large high schools who teach 
students from diverse ethnic and social class backgrounds might argue that 
working adults have more shared experience in common than adolescents 
coming from such diverse homes — and we regularly aggregate data from such 
students. 

The majority of data in this study comes from assessment questions that 
were exactly the same at all sites. Aggregating data from these items is relatively 
easy to justify. For items custom-designed for each program, the issue is less 
clear. The common framework of the evaluation model makes such items 
comparable in some sense, but there may be differences in difficulty level. For 
example, aggregating literacy performance data from several different custom- 
designed scenarios is problematical, but arguably acceptable when scores are 
standardized. Continued research in this area, beyond these beginning steps, is 
obviously needed. 

Allowing for all the above cautions and caveats, patterns across programs 
suggest some useful connections between the instructional uses of time and 
learner gains in a variety of literacy-related areas. The analysis of variance 
suggests that gains in (a) literacy performance on scenarios, (b) beliefs about 
personal effectiveness with literacy, and (c) clarity and definiteness of future 
educational plans are strongly linked to an environment intense with the use of 
workplace reading and writing materials, and providing regular discussion and 
feedback related to learner literacy processes, beliefs about personal literacy 
effectiveness, and future education^ plans. 
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It is conceivable and indeed likely that the focus and motivation provided 
by improved self-perceptions and aspirations will provide significant future 
benefits as learners choose additional educational experiences, attempt more 
literacy challenges, and slowly accrue more experience and practice with 
literacy. The improvement of literacy practice in the single 200-hour course 
suggests that such longer term gains are attainable, though not necessarily in 
brief, isolated programs. 

The brief duration of most workplace literacy programs and the limited 
degree of instructional transfer make it mandatory that program providers have 
clear goals for what they want to achieve in the limited time that learners are in 
class. Among the goals should be helping learners develop, through 
discussion and feedback, clearer senses of their own improving literacy 
abilities, broader senses of the literacy strategies available to them, and 
stronger links between what they are currently learning and future 
occupational and educational choices. Since time is so short in the courses, 
instructors should also be seeking ways to extend this time beyond the 
classroom. One way of doing this is to use on-the-job materials in class so 
that learners are more likely to continue practicing outside class time. Also, 
encouraging learner motivation and independence is likely to lead to learners 
engaging more often in literacy-related activities. 

Although data for this study were carefully gathered over a period of three 
years and across programs in several industries, the sm.all sample sizes 
available in most workplace literacy programs somewhat limit the findings. 
Findings are also limited by the fact that longitudinal follow-up of learners 
was not possible. As a result, questions about the impact of changed literacy 
beliefs and aspirations upon future literacy practices and abilities remain 
undocumented. 
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Appendix A 



Course Characteristics 



(Assessed by researchers for each course.) 

Time: 1 . Instructional time in hours — course meetings + homework 

2. Reading/writing intensity — 0-5 scale with descriptors 
Topic: 3. Workplace orientation— ^5 scale with descriptors 

4. Home/family orientation — 0-5 scale with descriptors 
Talk: 5. Discussion of literacy beliefs and plans — 0-5 scale with descriptors 

6. Discussion of reading and writing processes — 0-5 scale with descriptors 



Descriptor 

1. Instructional time in hours (not applicable) 

2. Reading/writing intensity 

Learner reading/writing: 

0 does not happen in this course 

1 happens occasionally/incidentally in this course 

2 happens sometimes in this course (20-30% of time) 

3 happens moderately often in this course (40-50% of time) 

4 happens often in this course (60-70% of time) 

5 happens very often in this course ( more than 80% of time) 

3. Workplace orientation 

Curriculum and materials: 

0 have no direct connection with the workplace 

1 use workplace examples occasionally 

2 use workplace examples sometimes (20-30% of time) 

3 use workplace examples much of the time (50-60% of time) 

4 are connected mainly to the workplace (70-80% of time) 

5 are connected entirely to the workplace (90-100% of time) 

4. Home/familv orientation 

Curriculum and materials: 

0 have no direct connection with the home/family 

1 use home/family examples occasionally 

2 use home/family examples sometimes (20-30% of time) 

3 use home/family examples much of the time (50-60% of time) 

4 are connected mainly to the home/family (70-80% of time) 

5 are connected entirely to the home/family (90-100% of time) 

5. Discussion of literacy beliefs and plans : 

0 does not occur in this course 

1 occurs occasionally/incidentally in this course 

2 occurs as a deliberate part of this course 

3 occurs moderately often in this course (every other session) 

4 occurs often in this course (most sessions) 

5 occurs very often in this course (every session) 
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6. Discussion of reading and writing processes : 

0 does not occur in this course 

1 occurs occasionally/incidentally in this course 

2 occurs as a deliberate part of this course 

3 occurs moderately often in this course (every other session) 

4 occurs often in thus course (most sessions) 

5 occurs very often in this course (every session) 
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Appendix B 



Learner Results: Areas of Gain 



For most areas, each learner’s gain score is the difference between the pretest and the 
posttest scores obtained from the sources described below. The exception is the use of 
holistic scores in the areas of Beliefs and Plans: Here the gain score is the sum of two or 
three holistic ratings each scored as 1 , 0, or -1 according to whether each learner showed 
gain, no change, or loss. (See Mikulecky & Lloyd, 1993, for details of the instruments 
used to obtain these responses.) 

1. Practices at Work 

Count of items reported in Interview as read or written at work in last week 

2. Practices Away From Work 

Count of items reported in Interview as read or written away from work in last 
week 

3. Reading Process 

Total count of strategies mentioned for reading workplace scenarios 
(most courses assessed with 3 scenarios, but some with 2 only) 

4. Scenario Performance 

Total score for workplace scenario content questions, with scores on each scenario 
standardized before being combined* 

(most courses assessed with 3 scenarios, but some with 2 only) 

5. Beliefs 

Sum of holistic ratings comparing pre- and post-responses for learners’ perception 
of literacy effectiveness and future literacy aspirations — comparisons based on 
learner self-rating comments and quality of reasons given for learner self- 
assessments (i.e., concrete mention of specific literacy abilities, activities, and 
interests) 

6. Plans 

Sum of holistic ratings comparing pre- and post-responses for 1-year plans, 5-year 
plans, and 10-year plans — comparisons based on clarity, definiteness, and detail of 
plans mentioned, particularly in relation to reading and education 
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* For each scenario, pretest and posttest scores were pooled and these were standardized as 
a single sample in order to avoid losing any differences between pretest and posttest 
scores — and hence any learner gains. 



The table below shows the mean, standard deviation, and range for each variable 
considered above. 



Variable 


Mean 


Standard 

Deviation 


Range 


Practices at work 


0.128 


2.459 


-10-7 


Practices away from 


0.251 


2.391 


-6-8 


work 


1.694 


4.365 


-9-24 


Reading process 


1.261 


2.168 


-4.2-7. 3 


Scenario performance 


0.452 


1.000 


-2-2 


Beliefs 


0.503 


1.432 


-3-3 


Plans 
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Appendix C 



Instruments Used to Gather Learner Data 

Note that parts of the instruments shown below were used at all sites, and parts were 
custom-designed for particular sites within an overall framework. For these latter parts, 
examples only are given — and are labeled as such. Note also that, to save space, the gaps 
for learner responses have been reduced from their original sizes. 

LEARNER INTERVIEW 

Personal Information: 

Name: Date: 

What class are you in? - 

Job you do 



I'd like to ask you some questions about reading, writing, and education. The answers to 
these questions will give us an idea of the way reading and writing are used here. 

Beliefs 

1. Describe someone you know who is good at reading and writing. What makes you 
choose this person? 



2. How good do you consider yourself to be at reading and writing? What makes you 
think so? 



3. Describe how you would like to be in terms of reading and writing. 
{Probe ; Could you give me some examples?) 



Practices 

1 . Tell me the sorts of things you read and write away from work during a normal week. 
{For probe, ask: "Can you give me more examples?") 



3 



C 

u 



NATIONAL CENTER ON ADULT LITERACY 



C - 



2. Tell me the sorts of things you read and write on the job during a normal week. 
{Use probe above for more examples.) 

Process : Article (Example from one site) 



1 . I am going to show you an article from the Employee Handbook. Explain to me how you 
would read this story in order to find out what the writer thinks. 

{Show attached article “A Message from The Chairman and CEO”.) 

Describe what you would look at. What would you be thinking about? How would you go 
about reading this story? What would you do first, then next, then next? 



2. (easy factual question) 

How many customers does the company serve? 



3. (harder factual question) 

Name one industry that uses the company’s products. 



4. (easy inference question) 

How does the company help to provide stable employment to workers? 



5. (harder inference question) 

How is the company’s leadership demonstrated? 



6. (harder application question) 

What prospects do you see for the growth of the company? 
Give reasons for your answer. 



7. (easy application question to end the section) 

As an employee how would the chairman’s message motivate you? 
Give reasons for your answer. 
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Our company is very fortunate to have almost 10,000 good 
customers throughout the United States, Canada, and more than 
30 other countries. These customers have been remarkably loyal 
throughout the years and we’re adding more every week. 

Our success depends to a great extent on our ability to please 
these customers. \^at pleases customers is good quality, on-time 
delivery, fair prices and fair treatment. We do our best to 
provide all of these and you are an important part of this. 

We serve many different markets, all the way from machine 
tool manufacturers, to farm machinery manufacturers, to 
distributors who supply replacement motors to industries of all 
kinds. This has helped us provide more stable employment than 
if we were to serve just one or two industries. 

While we supply only 4 or 5% of all the motors produced in 
the United States, we are a well-known manufacturer in this 
field. As a matter of fact, we’re one of the leaders in several 
ways. For example, much recognition has been achieved recently 
in the area of high-efficiency motors which not only conserve 
energy, but save considerable amounts of money for people 
buying and using our motors. 

We want you to be proud of your company, as I am. We 
welcome your ideas, your support and your helping to produce 
more and better motors. 



Sincerely 



Chairman and CEO 
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Process : Procedure/Job Aid (Example from one site) 



1 . I am going to show you an instruction sheet. Explain to me how you would read it in 
order to find out what it’s about. 

{Show attached instruction sheet, “Hand Washing”.) 

Describe what you would look at. What would you be thinking about? How would 
you go about reading this instruction sheet? What would you do first, then next, then 
next? 



2 (easy factual question) 

What should you do if your skin becomes dry and infected after washing 
your hands frequently? 



3. (harder factual question) 

What kind of soap must be in a container that will drain? 



4. (easy inference) 

Look at drawings 4, 5 and 6. What is the point of the drawings? 



5. (harder inference) 

What are the reasons for using paper towels on the hand washing 
instruction sheet? 



6. (harder application) 

Why does it say in step #1 of the Procedure, DO NOT FILL SINK? 



7. (easy application question to end the section) 

Explain two jobs which you perform that require hand washing before 
or after the job. 



4l 
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X, lUfFECTtOf? COWTROL 

G. handwashing 



i. 



2 . 



3 . 



I*. 



5 . 



6. 



1. 



2 , 



3 . 






3 . 






T. 



8. 



J. 

10 . 



Required $q)pl{i8 and Equipment: 

Hand Brush, Paper TdvrtU.Saap (ter, Uqu;d, povder), Running vater 



Safety PreciutionM PROCEDyRE: 

Alv^ys use paper lowcls to turn iauceu cn 
and of{« 

If ycu have any cuta or other ^hln breaks you 
must wear gloves. 

Skin may become dry and may become 
infectedr because of frequent hand washing, 
thertofere, apply lotion after washing. 

Follow hand washing procedure step by step 
i* do not take any short cuts* 




Where bar soap is usedi It must be in a 
container that will drain, rinse soap off 
before using* 

Use clean brush. 

Prooaduret 

Take paper towel and turn on faucet. 

DO NOT FILL SWK. 

Wet hands* 

Apply soap tharooghiy ^ get imder nail and 
between lingers* 

Use ciMR brush to remove any substances 
offering parcioalar feststance^ 

Rub hands together. Use a routing 
frictional motion* Count to JO. 




Interlace the flitgers. Rub up arid down. 

Rinse will under running water. 

Dry with paper towel, 

Turn off faucet, using paper towel 

Open door with paper towel before 
diagarding towel, 




\ 
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Process : Graph (Example from one site) 



1 . I am going to show you a graph. Explain to me how you would read this graph in order 
to find out what it's about. 

{Show attached graph. “Payment Scores vs Class Average”). 

Describe what you would look at. What would you be thinking about? How would 
you go about reading this graph? What would you do first, then next, then next? 



2. (easy factual question) 

What is the graph about? 



3. (harder factual question) 

What comparison is being made on the graph? 



4. (easy inference) 

In how many areas is Lisa higher than the class? 



5. (harder inference) 

Overall in what areas does Lisa need to improve her payments most? 



6. (easy application question) 

Is Lisa more knowledgeable about some insurance benefits than others? 
Give reasons for your answer. 



7. (more difficult application question) 

Which areas are you best at? Give reasons why you think so. 
Which areas are worst for you? Why do you think so? 
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"PAYMENT" Scores vs Class Average 
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Plans 



Now I'd like to ask you about your plans. Explain how you see reading and edu cation as 
part of these plans: 

1 . What are your plans for the next year? 



2. What are your plans for the next 5 Years? 



3. What are your plans for the next 10 Years? 




c - vlii 



TECHNICAU REPORT TR96-03 



O 




LEARNER QUESTIONNAIRE 

Name: Age: Sex: 

Education: (furthest year in school) Training 

Marriage Status: 

Children: (number) (ages) 

Practices: Self rating reading ability 

1 . 1. First check only the things you've read in the past month . 

2 . Now go back and rate your ability to read the items you've checked. 



poor 

local newsoapers 


1 2 


3 


4 


excellent 

5 


classified ads 


1 2 


3 


4 


5 


teleohone bills 


1 2 


3 


4 


5 


TV guide listinas 


1 2 


3 


4 


5 


magazines 


1 2 


3 


4 


5 


poor 








excellent 


training guides 


1 2 


3 


4 


5 


pavcheck stubs 


1 2 


3 


4 


5 


companv newsletters 


1 2 


3 


4 


5 


benefit information 


1 2 


3 


4 


5 


graohs and charts 


1 2 


3 


4 


5 


(For items 11-15, workplace personnel chose appropriate items of reading material. 


following are examples only) 










poor 








excellent 


procedure manual 


1 2 


3 


4 


5 


inter-office memos 


1 2 


3 


4 


3 


notes from supervisor 


1 2 


3 


4 


S 


computer screens 


1 2 


3 


4 


5 


computer printouts 


J 2_ 


3 


4 


5 
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Practices: Reading frequency 

Please check the number of times you have done the following: 

1. In the last 7 days how many times have you used a TV guide listing to select 
programs? 

0 i 2 3 4 5 6 7 8 9 Wh 

2. In the last 7 days how many times have you read a newspaper? 

0 i 2 3 4 5 6 7 8 9 

3. In the last 7 days how many times have you read a magazine? 

0 i 2 3 4 5 6 7 8 9 1^ 

4. In the last 7 days how many times have you read a book for pleasure? 

0 i 2 3 4 5 6 7 8 9 

5. In the last 7 days how many times have you read the following types of books? 



mystery: 


times 


how-to books: _ 


_ times 


novels: 


times 


factual books: _ 


_ times 


poetry: 


times 


encyclopedia: _ 


_ times 


Bible: 


times 


comic books: 


_ times 



other types: times 

times 



6. How often do you make a shopping list before you go to the store? 
never occasionally often always 



7. When you're waiting in an office, how often do you read magazines? 
never occasionally often always 



8. Do you subscribe to any magazines? yes no 
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If yes, which ones? 

9. How many different magazine titles do you have in your home? 

0 i 2 3 4 5 6 7 8 9 

10. How many books are in your home, either owned or borrowed? 

0 10 20 30 40 50 60 70 80 90 1^ 

Practices: Literacy at work 

Please circle the number which best describes you in the situations below; 

(1) You just listen in team or department meeting discussions. 

very like me 1 2 3 4 5 very unlike me 

(2) You talk a lot in team or department meetings, asking questions or sharing ideas. 

very like me 1 2 3 4 5 very unlike me 

(3) Your ideas are often discussed in team or department meetings. 

very like me 1 2 3 4 5 very unlike me 

(4) You wait for others to talk about written information, just to be sure what is in it. 

very like me 1 2 3 4 5 very unlike me 

(5) You look for printed directions to help figure out what to do when a problem arises. 

very like me 1 2 3 4 5 very unlike me 

(6) You often have trouble reading paperwork from management. 

very like me 1 2 3 4 5 very unlike me 




(7) When the booklet about new health benefits arrived, you read it carefully, 
very like me 1 2 3 4 5 very unlike me 
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Practices : Family Literacy 



Only answer the following questions if you have a child between the ages 
of 3-17 at home. 

Please answer for your youngest child in this age group and please fill in only one answer 
per question: 



1. This child is years old. 

2. In the last 7 days how many times has your child looked at or read books or 
magazines? 

0 i 2 3 4 5 6 7 8 9 1^ 



3. In the last 7 days how many times has your child seen you readi ,g or writing? 
0 i 2 3 4 5 6 7 8 9 1^ 



4. In the last 7 days how many times have you helped your child with homework and/or 
with school projects? 



0123456789 10+ 



5. In the last 7 days how many times have you read/looked at books with your child or 
listened to him/her read? 



0123456789 10+ 



6. In the last 7 days how many times has your child asked to be read to? 

0 i 2 3 4 5 6 7 8 9 1^ 



7. In the last 7 days how many times has your child printed, made letters, or written? 

0 i 2 3 4 5 6 7 8 9 1^ 



8. In the last month how many times has your child gone to a public library? 
0 i 2 3 4 5 6 7 8 9 



9 . In the last month how many times have you participated/helped out in your child's 
school? 



0 1 23456789 10+ 

50 
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LEARNER QUESTIONNAIRE (addendum) 



For Year 3 of the project, the questionnaire was revised in order to evaluate learner 
practices better. The following questions replaced those above on practices at work and 
practices away from work. 



Plea.se check the number of times you have done the following: 

1 . In the last 7 days how many times have you read a company newsletter or bulletin 
board? 



012345678 
2. In the last 7 days how many times have you used a manual? 


9 


10+ 


012345678 
3. In the last 7 days how many times have you read a memo or letter? 


9 


10+ 


012345678 
4. In the last 7 days how many times have you used a graph or chart? 


9 


10+ 


012345678 
5. In the last 7 days how many times have you used an instruction sheet? 


9 


10+ 


0123456789 
6. In the last 7 days how many times have you written a note to a co-worker? 


10+ 


012345678 


9 


10+ 



(For questions 7-10, workplace personnel chose appropriate items of reading material. 
The following items are examples only) 



7. In the last 7 days how many times have you used a job order form ? 



012345678 
8. In the last 7 days how many times have you used a route sheet ? 


9 


10+ 


012345678 
9. In the last 7 days how many times have you used a computer screen ? 


9 


10+ 


012345678 
10. In the last 7 days how many times have you used a blueprint ? 


9 


10+ 


012345678 


9 


10+ 
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13. How often do you have trouble reading paperwork from management? 

never occasionally often always 

14. When a problem arises, how often do you look for printed directions to help figure out 
what to do? 

never occasionally often always 



15. How often are you able to help when someone has trouble reading something? 
never occasionally often always 



16. How often do you talk in team or department meetings, asking questions or sharing 
ideas? 

never occasionally often always 



17. How often are your ideas discussed in team or department meetings? 
never occasionally often always 
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Questionnaire: Practices away from work 

Please check the number of times you have done the following: 

1 . In the last 7 days how many times have you read a newspaper? 

0 i 2 3 4 5 6 7 8 9 1^ 

2. In the last 7 days how many times have you read mail, bills or ads? 

0 i 2 3 4 5 6 7 8 9 1^ 

3. In the last 7 days how mai.y times have you read a magazine? 

0 i 2 3 4 5 6 7 8 9 1^ 

4. In the last 7 days how many times have you read a book for pleasure? 

0 i 2 3 4 5 6 7 8 9 1^ 

5. In the last 7 days how many times have you read in order to dQ something? 

(For example: buy, build, cook, fix) 

0 i 2 3 4 5 6 7 8 9 1^ 

6. In the last 7 days how many times have you discussed something you've read with 

another person? 

0 i 2 3 4 5 6 7 8 9 1^ 

7. In the last 7 days how many times have you written notes to people you live with? 

0 i 2 3 4 5 6 7 8 9 1^ 

8. In the last 7 days how many books have you brought into your home, either bought or 
borrowed? 



0123456789 10+ 



9. In the last 7 days how many times have you read the following types of books ? 

mystery: times how-to books: times 

novels: times factual books: times 

poetry: times encyclopedia: times 

Bible: times comicbooks: fmes 

other types: times times 



10. When you're waiting in an office, how often do you read magazines? 

never occasionally often always 

1 1 . How often do you make a shopping list before you go to the store? 

never occasionally often always 

54 
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CLOZE EXERCISE (Example from one site) 
Name Date 



In a cloze exercise, you try to guess which words are missing. For example, in the sentence 
below, a word is missing. 



She looked before she 



the street. 



A good guess for the missing word is "crossed." 

She looked before she . crossed the street. 

In the story below, try to guess and replace the missing words. Don't expect to get them all. 
Some are nearly impossible. 

G.M Designs Safety for All Ages 

We all like to think about the old days. Life seemed simpler and, in some ways, better 

then. But when it comes to , the good old days offer the 

same degree safety as today's cars trucks. Advancements 



in technology 
safest 



the G.M. vehicle you 



. today among the 



the world. Each G.M. 



thousands of dedicated men 
safety of their customers. 



and truck is backed 
women who care about 



themselves. 



have a stake in 



., as G.M. customers 
_ G.M. vehicles the liighest 



quality and reliability. 

you're wondering if safety 



improved in recent 



this: The classic 1955 



would require more than 



changes 



major changes or additions 
be as safe as 



hundreds of incremental 



vehicles. 



From: Kilborn, C. GM Today (November/December. 1990). page 1. 
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EMPLOYEE ASSESSMENT - OVERALL RATING 

(Examples from several sites) 

Please rate each employee on a scale of 1 - 10 for each aspect below. 

• An average employee would be rated 5. 

• A top employee would be rated 8 or higher. 

• A bottom employee would be rated 2 or lower. 



EMPLOYEE DATE 



RATER 



PAPERWORK 








Bottom 


Averaee 


Tod 




intimidated by job-related 
paperwork and does it 
poorly 


does job-related paperwork, 
simply keeping pace 


completes all job-related 
paperwork and tries to 
improve procedures 




12 3 


4 5 6 7 


8 9 


10 


MACHINE SETTING 








Bottom 


Average 


Top 




unable to set machines 
correctly 


usually sets machines 
correctly, but doesn't always 
check settings 


sets machines correctly and 
checks settings thoroughly 


1 2 3 


4 5 6 7 


8 9 


10 


PROCEDURES 








Bottom 


Average 


Im 




does not follow procedure in 
a step-by-step process, 
sometimes missing areas of 
the Job 


follows procedure most of 
the time, occasionally 
missing a step 


follows step-by-step 
procedure 





123456789 10 





C • xvtD 



TECHNICAL REPORT TR86-03 



I 



SAFETY PRECAUTIONS 



H Bottom 


Average 


lOR 


B 

has to be reminded 
about safety and does 
B not use caution on the 

B job 


takes precautions, 
but sometimes 
overlooks small 
areas on the job 


follows safety precautions 
and understands why 


1 12 3 


4 5 6 


7 8 9 10 


1 PROBLEM-SOLVING 






Bottom 

H| 


Average 


Top 


B calls supervisor on minor 

details or continues to 

B work when equipment is 

1 faulty 


makes minor adjustments, 
offers solutions to 
problems and calls 
supervisor only when 
necessary 


can analyze job situations, 
make suggestions and 
solutions which implement 
change 


1 12 3 


4 5 6 


7 8 9 10 


1 COMMUNICATION 






n Bottom 


Average 


Top 


B 

doesn’t talk about the job, 
doesn’ t speak at department 
1 H meetings 


talks about the job and offers talks about the job, has 
suggestions at department suggestions and shows 

meetings leadership 


12 3 

1 


4 5 6 


7 8 9 10 


OUALITY OF PAPERWORK 




j B Bottom 


Average 


Top 


1 m paperwork provides 

1 B no or limited 

1 information; 

1 ■ illegible, poor 

I B grammar 


paperwork usually 
acceptable; at times too 
brief or vague 


paperv/ork is legible, 
detailed, clear and 
concise 


1 12 3 


4 5 6 


7 8 9 10 
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